{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 基于CRF的医疗实体识别\n",
    "\n",
    "## 作业要求\n",
    "在这个项目中，我们对医疗文本做实体识别，这是一个经典的序列标注问题， 我们需要对文本里的每一个字做实体的标注，将使用两种不同的方法：\n",
    "- ```特征工程+CRF```： 这个方法针对于每一个字抽取一些特征，如这个字属于哪一个单词，这个字的前面一个字是什么，后面的字是什么等等。做完特征工程之后，我们就有了针对于每一个字的特征向量，之后把这些特征向量作为CRF的输入，并训练模型。\n",
    "- ```利用LSTM-CRF```：这个方法可以认为，特征工程的部分由LSTM来做，所以我们在这个方法论下不需要做特征工程，所有特征是自动被LSTM模块学出来的，剩下的CRF部分保持不变。\n",
    "\n",
    "你需要在标记为```TODO```的地方填写代码即可。 \n",
    "\n",
    "## 文件说明\n",
    "* 基于CRF的医疗实体识别.ipynb：主脚本。\n",
    "* 基于CRF的医疗实体识别.pdf：操作手册。\n",
    "* 数据源：[基于CRF的医疗实体识别(阿里云盘)](https://www.aliyundrive.com/s/eKEX7JbJyCr)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#导入数据分析所需的基础包\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "#包tqdm是用来对可迭代对象执行时生成一个进度条用以监视程序运行过程\n",
    "from tqdm import tqdm\n",
    "\n",
    "#导入训练集测试集划分的包\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#导入CRF模型所需的包\n",
    "# !pip install sklearn_crfsuit\n",
    "# 安装语句实际上是中划线https://sklearn-crfsuite.readthedocs.io/en/latest/install.html\n",
    "from sklearn_crfsuite import CRF\n",
    "\n",
    "#导入模型评估所需的包\n",
    "from sklearn_crfsuite import metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 读取数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据保存在data/目录下，data/目录下共有四个文件夹，分别对应四种医学情景：出院情况、病史特点、诊疗过程和一般项目。每个文件夹下保存了该情景下的电子病历。包括两类文件：'xxx-yyy.txtoriginal.txt'和'xxx-yyy.txt'。'xxx-yyy.txtoriginal.txt'保存了xxx情境下第yyy号病历的病历文本，保存在txt的第一行中。'xxx-yyy.txt'为其对应的标签数据。\n",
    "\n",
    "数据中共包含5种实体：治疗、身体部位、疾病和诊断、症状和体征、检查和检验。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "女性，88岁，农民，双滦区应营子村人，主因右髋部摔伤后疼痛肿胀，活动受限5小时于2016-10-29；11：12入院。\n"
     ]
    }
   ],
   "source": [
    "# 读取一个病历文本数据，并查看其内容。\n",
    "with open('data/一般项目/一般项目-1.txtoriginal.txt') as f:\n",
    "    content = f.read().strip()\n",
    "print(content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "右髋部\t21\t23\t身体部位\n",
      "疼痛\t27\t28\t症状和体征\n",
      "肿胀\t29\t30\t症状和体征\n"
     ]
    }
   ],
   "source": [
    "# 读取上述病历对应的标签数据\n",
    "with open('data/一般项目/一般项目-1.txt') as f:\n",
    "    content_label = f.read().strip()\n",
    "print(content_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出，标签文件的数据格式为每行对应一个实体，每行格式为“实体内容 实体在文本中的开始位置 实体在文本中的结束位置 实体类别”。如第一行表示content[21:24]对应的便是'右髋部'，为身体部位实体类别。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据标注"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实体识别的数据标注方式主要有BIOES和BIO两种，详细的介绍参考实验手册。这里为使标注类别不至于太多，我们采用BIO方式。即将实体部分的第一个汉字标注为B，实体的其他部分的汉字标注为I，非实体部分标注为O。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将5种实体类别治疗、身体部位、疾病和诊断、症状和体征、检查和检验分别标记为TREATMENT、BODY、DISEASES、SIGNS、EXAMINATIONS。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "则标记时，如：若为治疗类别的实体的第一个汉字，则将其标注为B-TREATMENT，该实体其他字标记为I-TREATMENT。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_dict = {'治疗':'TREATMENT',\n",
    "              '身体部位':'BODY',\n",
    "              '疾病和诊断':'DISEASES',\n",
    "              '症状和体征':'SIGNS',\n",
    "              '检查和检验':'EXAMINATIONS'}\n",
    "\n",
    "def sentence2BIOlabel(sentence,label_from_file):\n",
    "    '''\n",
    "        返回句子sentence的BIO标注列表\n",
    "        入参：\n",
    "            sentence：一个句子，字符串类别\n",
    "            label_from_file：该句子对应的标签，格式为直接从txt文件中读出的格式，形如上文中的content_label\n",
    "        出参：\n",
    "            sentence_label：该句子的BIO标签。一个列表，列表的第i项为第i个汉字对应的标签\n",
    "    '''\n",
    "    #初始的sentence_label每个标签均定义为'O'。之后会修改其中实体部分的标签。\n",
    "    sentence_label = ['O']*len(sentence)\n",
    "    if label_from_file=='':\n",
    "        return sentence_label\n",
    "    \n",
    "    #line为label_from_file中每一行的数据，对应一个实体的信息。格式为“实体内容 实体在文本中的开始位置 实体在文本中的结束位置 实体类别”\n",
    "    for line in label_from_file.split('\\n'):\n",
    "        #entity_info中保存了单个实体的信息\n",
    "        entity_info = line.strip().split('\\t')\n",
    "        start_index = int(entity_info[1])     #实体在文本中的开始位置\n",
    "        end_index = int(entity_info[2])      #实体在文本中的结束位置\n",
    "        entity_label = label_dict[entity_info[3]]      #实体标签类别\n",
    "        #为实体的第一个汉字标记为B-xx\n",
    "        sentence_label[start_index] = 'B-'+entity_label\n",
    "        #为实体中的其他汉字标记为I-xx\n",
    "        for i in range(start_index+1,end_index+1):\n",
    "            sentence_label[i] = 'I-'+entity_label\n",
    "    return sentence_label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-BODY', 'I-BODY', 'I-BODY', 'O', 'O', 'O', 'B-SIGNS', 'I-SIGNS', 'B-SIGNS', 'I-SIGNS', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "# 以上文中的content和content_label为例查看sentence2BIOlabel函数的使用方法\n",
    "# 返回上文中content对应的BIO标签并输出\n",
    "sentence_label_tmp = sentence2BIOlabel(content,content_label)\n",
    "print(sentence_label_tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "女 O\n",
      "性 O\n",
      "， O\n",
      "8 O\n",
      "8 O\n",
      "岁 O\n",
      "， O\n",
      "农 O\n",
      "民 O\n",
      "， O\n",
      "双 O\n",
      "滦 O\n",
      "区 O\n",
      "应 O\n",
      "营 O\n",
      "子 O\n",
      "村 O\n",
      "人 O\n",
      "， O\n",
      "主 O\n",
      "因 O\n",
      "右 B-BODY\n",
      "髋 I-BODY\n",
      "部 I-BODY\n",
      "摔 O\n",
      "伤 O\n",
      "后 O\n",
      "疼 B-SIGNS\n",
      "痛 I-SIGNS\n",
      "肿 B-SIGNS\n",
      "胀 I-SIGNS\n",
      "， O\n",
      "活 O\n",
      "动 O\n",
      "受 O\n",
      "限 O\n",
      "5 O\n",
      "小 O\n",
      "时 O\n",
      "于 O\n",
      "2 O\n",
      "0 O\n",
      "1 O\n",
      "6 O\n",
      "- O\n",
      "1 O\n",
      "0 O\n",
      "- O\n",
      "2 O\n",
      "9 O\n",
      "； O\n",
      "1 O\n",
      "1 O\n",
      "： O\n",
      "1 O\n",
      "2 O\n",
      "入 O\n",
      "院 O\n",
      "。 O\n"
     ]
    }
   ],
   "source": [
    "# 输出content中每个汉字与BIO标签的对应关系\n",
    "for i in range(len(content)):\n",
    "    print(content[i],sentence_label_tmp[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```TODO1``` 循环每个样本文件，把样本统一存放在sentence_list中，同时将样本对应的BIO标签存放在label_list变量中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "入院后完善相关检验、检查。明确患者诊断，给予应用止疼、活血化瘀、营养脑神经等药物对症治疗。后请康复科会诊。建议患者康复治疗。现患者自觉症状好转，要求出院，今日给予办理出院手续。\n",
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'I-TREATMENT', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "# sentence_list的格式为[第一个句子，第二个句子，第三个句子，...，第n个句子]\n",
    "# label_list的格式为[第一个句子对应的BIO标注列表，第二个句子对应的BIO标注列表，第三个句子对应的BIO标注列表，...，第n个句子对应的BIO标注列表]\n",
    "sentence_list = []\n",
    "label_list = []\n",
    "\n",
    "## 在这里输入你的代码\n",
    "## 获取文件位置\n",
    "import os\n",
    "file_dir='data'\n",
    "for root, dirs, files in os.walk(file_dir): \n",
    "    for name in files:\n",
    "        f_name=os.path.join(root, name)\n",
    "        ##以origin文件作为基础，先处理句子内容列表\n",
    "        if 'txtoriginal.txt' in f_name:\n",
    "            #print(f_name)\n",
    "            with open(f_name) as f:\n",
    "                f_content = f.read().strip()\n",
    "            sentence_list.append(f_content)\n",
    "            ##再处理BIO标注列表\n",
    "            l_name=f_name.replace('txtoriginal.txt','txt')\n",
    "            with open(l_name) as f:\n",
    "                l_content = f.read().strip()\n",
    "            label_list.append(sentence2BIOlabel(f_content,l_content))\n",
    "##查看结果\n",
    "print(sentence_list[0])\n",
    "print(label_list[0])\n",
    "## 结束你的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本特征工程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要使用CRF算法对每个字进行标注，就需要获取每个字对应的特征。就需要对文本进行特征工程，这一部分就是构建一句话中每个字的特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: \"https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/medicine.zip\" to /Users/zhuochen/.pkuseg/medicine.zip\n",
      "100%|██████████| 48189165/48189165 [00:10<00:00, 4671245.78it/s]\n",
      "Downloading: \"https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/postag.zip\" to /Users/zhuochen/.pkuseg/postag.zip\n",
      "100%|██████████| 41424981/41424981 [00:05<00:00, 7567284.60it/s]\n"
     ]
    }
   ],
   "source": [
    "#中文分词时最常用的包是jieba。但我们本次的数据集是专门针对医疗领域的，这里选用了一个在细分领域上表现更好的库pkuseg\n",
    "import pkuseg\n",
    "# 将model_name设置为'medicine'以加载医疗领域的模型。第一次执行此代码时会自动下载医疗领域对应的模型，这可能需要一些时间。\n",
    "# 设置postag为True会在分词的同时进行词性标注\n",
    "seg = pkuseg.pkuseg(model_name='medicine',postag=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('发病', 'vn'),\n",
       " ('原因', 'n'),\n",
       " ('为', 'v'),\n",
       " ('右髋部', 'n'),\n",
       " ('摔伤', 'v'),\n",
       " ('后', 'f'),\n",
       " ('疼痛', 'a'),\n",
       " ('肿胀', 'v')]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pkuseg包使用示例\n",
    "seg.cut('发病原因为右髋部摔伤后疼痛肿胀')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出，pkuseg包对医疗方面的文本有较好的分词效果。seg.cut(文本)的输出格式为[(第一个词，第一个词的词性),(第二个词，第二个词的词性),...,(第n个词，第n个词的词性)]。稍后在构建每个字的特征时我们会用到pkuseg的分词功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 加载医学的专业词汇词库THUOCL_medical.txt。这一文件是从https://github.com/thunlp/THUOCL中下载而来。\n",
    "# 文件中每行的格式为：医学名词 词频\n",
    "\n",
    "# 读取文件\n",
    "with open('THUOCL_medical.txt') as f:\n",
    "    medical_words = f.read().strip()\n",
    "# 获取医疗词汇表\n",
    "medical_words_list = [words.strip().split('\\t')[0] for words in medical_words.split('\\n')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['精神', '医院', '检查', '死亡', '恢复', '意识', '医疗', '治疗', '卫生', '患者']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#医疗词汇表示例，这一词汇表在我们构建特征时会用到。\n",
    "medical_words_list[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "进行完上述准备工作后，我们接下来正式来构造特征。\n",
    "\n",
    "```TODO2``` 你需要完成部分特征提取的代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word2feature(sentence,i):\n",
    "    '''\n",
    "        返回句子sentence中第i个汉字的一些简单的特征\n",
    "        入参：\n",
    "            sentence：待处理的句子\n",
    "            i：会返回第i个汉字的一些简单的特征\n",
    "        出参：\n",
    "            simple_feature：由一些简单的特征所组成的字典，字典的键为特征名，值为特征值\n",
    "    '''\n",
    "    simple_feature = {}\n",
    "    simple_feature['word'] = sentence[i]   #当前字\n",
    "    simple_feature['pre_word'] = sentence[i-1] if i>0 else 'start'    #前一个字\n",
    "    simple_feature['after_word'] = sentence[i+1] if i<len(sentence)-1 else 'end'     #后一个字\n",
    "    #接下来加入当前字的Bi-gram特征，即前一个字+当前字、当前字+后一个字，并将特征分别命名为'pre_word_word'和'word_after_word'\n",
    "    ##在这里输入你的代码\n",
    "    simple_feature['pre_word_word']=simple_feature['pre_word']+simple_feature['word']\n",
    "    simple_feature['word_after_word']=simple_feature['word']+simple_feature['after_word']\n",
    "    ##结束你的代码\n",
    "    simple_feature['bias'] = 1\n",
    "    return simple_feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence2feature(sentence):\n",
    "    '''\n",
    "        在word2feature定义的简单特征的基础上，增加一些复杂的特征，并返回句子中每个字对应的特征字典所组成的列表\n",
    "        入参：\n",
    "            sentence:待处理的句子\n",
    "        出参：\n",
    "            sentence_feature_list:句子中每个字对应的特征字典所组成的列表。格式为：[第一个字的特征字典，第二个字的特征字典，...，第n个字的特征字典]\n",
    "    '''\n",
    "    # 首先获取简单的特征\n",
    "    sentence_feature_list = [word2feature(sentence,i) for i in range(len(sentence))]\n",
    "    \n",
    "    # 接下来，为每个字抽取一些复杂的特征\n",
    "    # 增加当前字在分词后所在的词，该词的词性，该词的上一个词，该词的下一个词，该词是否为医疗专业词汇，该字是否为该词的第一个字\n",
    "    \n",
    "    word_index = 0    # 指向字的指针，会逐步往后移动，其作用在之后可以看到\n",
    "    # 使用pkuseg对句子进行分词\n",
    "    sentence_cut = seg.cut(sentence)\n",
    "    # 这里为和字进行区分，使用大写的WORD来表示词，小写的word来表示字\n",
    "    for i,(WORD,nominal) in enumerate(sentence_cut):\n",
    "        for j in range(word_index,word_index+len(WORD)):\n",
    "            sentence_feature_list[j]['WORD'] = WORD     # 当前字在分词后所在的词\n",
    "            sentence_feature_list[j]['nominal'] = nominal     # 该词的词性\n",
    "            sentence_feature_list[j]['pre_WORD'] = sentence_cut[i-1][0] if i>0 else 'START'    # 该词的上一个词\n",
    "            sentence_feature_list[j]['after_WORD'] = sentence_cut[i+1][0] if i<len(sentence_cut)-1 else 'END'    # 该词的下一个词\n",
    "            sentence_feature_list[j]['is_medicalwords'] = 1 if WORD in medical_words_list else 0    # 该词是否为医学专业词汇\n",
    "        # 加入一个特征'is_first'表示当前字是否为其所属词的第一个字，是则将该特征值记为1，否则记为0\n",
    "        \n",
    "        ## 在这里输入你的代码\n",
    "            sentence_feature_list[j]['is_first'] = 1 if sentence_feature_list[j]['word'] in WORD[0] else 0 \n",
    "        \n",
    "        \n",
    "        ## 结束你的代码\n",
    "        \n",
    "        word_index = word_index+len(WORD)    # 更新word_index的值\n",
    "    return sentence_feature_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1198/1198 [02:35<00:00,  7.70it/s]\n"
     ]
    }
   ],
   "source": [
    "#获取sentence_list中每句话中每个字对应的特征，并将其保存在feature_list中\n",
    "#使用tqdm函数来输出一个进度条，以监控代码的运行过程\n",
    "feature_list = [sentence2feature(sentence) for sentence in tqdm(sentence_list)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CRF模型搭建"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先对数据划分训练集和测试集\n",
    "x_train,x_test,y_train,y_test = train_test_split(feature_list, label_list, test_size=0.3, random_state=2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'CRF' object has no attribute 'keep_tempfiles'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/IPython/core/formatters.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, obj, include, exclude)\u001b[0m\n\u001b[1;32m    968\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    969\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mmethod\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 970\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minclude\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minclude\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexclude\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexclude\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    971\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    972\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/base.py\u001b[0m in \u001b[0;36m_repr_mimebundle_\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m    462\u001b[0m         \u001b[0;34m\"\"\"Mime bundle used by jupyter kernels to display estimator\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    463\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m\"text/plain\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mrepr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 464\u001b[0;31m         \u001b[0;32mif\u001b[0m \u001b[0mget_config\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"display\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'diagram'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    465\u001b[0m             \u001b[0moutput\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"text/html\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mestimator_html_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    466\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/base.py\u001b[0m in \u001b[0;36m__repr__\u001b[0;34m(self, N_CHAR_MAX)\u001b[0m\n\u001b[1;32m    258\u001b[0m                 \u001b[0mvalid_params\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    259\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 260\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msub_params\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnested_params\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    261\u001b[0m             \u001b[0mvalid_params\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0msub_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/pprint.py\u001b[0m in \u001b[0;36mpformat\u001b[0;34m(self, object)\u001b[0m\n\u001b[1;32m    142\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mpformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m         \u001b[0msio\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_StringIO\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 144\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msio\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    145\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0msio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    146\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/pprint.py\u001b[0m in \u001b[0;36m_format\u001b[0;34m(self, object, stream, indent, allowance, context, level)\u001b[0m\n\u001b[1;32m    159\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_readable\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    160\u001b[0m             \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 161\u001b[0;31m         \u001b[0mrep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlevel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    162\u001b[0m         \u001b[0mmax_width\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_width\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mindent\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mallowance\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    163\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrep\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0mmax_width\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/pprint.py\u001b[0m in \u001b[0;36m_repr\u001b[0;34m(self, object, context, level)\u001b[0m\n\u001b[1;32m    391\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlevel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    392\u001b[0m         repr, readable, recursive = self.format(object, context.copy(),\n\u001b[0;32m--> 393\u001b[0;31m                                                 self._depth, level)\n\u001b[0m\u001b[1;32m    394\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mreadable\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    395\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_readable\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/utils/_pprint.py\u001b[0m in \u001b[0;36mformat\u001b[0;34m(self, object, context, maxlevels, level)\u001b[0m\n\u001b[1;32m    179\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    180\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_changed_only\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 181\u001b[0;31m             \u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_changed_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    182\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    183\u001b[0m             \u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdeep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/utils/_pprint.py\u001b[0m in \u001b[0;36m_safe_repr\u001b[0;34m(object, context, maxlevels, level, changed_only)\u001b[0m\n\u001b[1;32m    423\u001b[0m         \u001b[0mitems\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpprint\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_safe_tuple\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    424\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mitems\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 425\u001b[0;31m             krepr, kreadable, krecur = saferepr(\n\u001b[0m\u001b[1;32m    426\u001b[0m                 k, context, maxlevels, level, changed_only=changed_only)\n\u001b[1;32m    427\u001b[0m             vrepr, vreadable, vrecur = saferepr(\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/utils/_pprint.py\u001b[0m in \u001b[0;36m_changed_params\u001b[0;34m(estimator)\u001b[0m\n\u001b[1;32m     89\u001b[0m     estimator with non-default values.\"\"\"\n\u001b[1;32m     90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 91\u001b[0;31m     \u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdeep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     92\u001b[0m     \u001b[0mfiltered_params\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     93\u001b[0m     init_func = getattr(estimator.__init__, 'deprecated_original',\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/base.py\u001b[0m in \u001b[0;36mget_params\u001b[0;34m(self, deep)\u001b[0m\n\u001b[1;32m    193\u001b[0m         \u001b[0mParameters\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    194\u001b[0m         \u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 195\u001b[0;31m         \u001b[0mdeep\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdefault\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    196\u001b[0m             \u001b[0mIf\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwill\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mparameters\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mthis\u001b[0m \u001b[0mestimator\u001b[0m \u001b[0;32mand\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    197\u001b[0m             \u001b[0mcontained\u001b[0m \u001b[0msubobjects\u001b[0m \u001b[0mthat\u001b[0m \u001b[0mare\u001b[0m \u001b[0mestimators\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'CRF' object has no attribute 'keep_tempfiles'"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'CRF' object has no attribute 'keep_tempfiles'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/IPython/core/formatters.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, obj)\u001b[0m\n\u001b[1;32m    700\u001b[0m                 \u001b[0mtype_pprinters\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtype_printers\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    701\u001b[0m                 deferred_pprinters=self.deferred_printers)\n\u001b[0;32m--> 702\u001b[0;31m             \u001b[0mprinter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpretty\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    703\u001b[0m             \u001b[0mprinter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflush\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    704\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mstream\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/IPython/lib/pretty.py\u001b[0m in \u001b[0;36mpretty\u001b[0;34m(self, obj)\u001b[0m\n\u001b[1;32m    392\u001b[0m                         \u001b[0;32mif\u001b[0m \u001b[0mcls\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    393\u001b[0m                                 \u001b[0;32mand\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'__repr__'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 394\u001b[0;31m                             \u001b[0;32mreturn\u001b[0m \u001b[0m_repr_pprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcycle\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    395\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    396\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0m_default_pprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcycle\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/IPython/lib/pretty.py\u001b[0m in \u001b[0;36m_repr_pprint\u001b[0;34m(obj, p, cycle)\u001b[0m\n\u001b[1;32m    698\u001b[0m     \u001b[0;34m\"\"\"A pprint that just redirects to the normal repr function.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    699\u001b[0m     \u001b[0;31m# Find newlines and replace them with p.break_()\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 700\u001b[0;31m     \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrepr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    701\u001b[0m     \u001b[0mlines\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplitlines\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    702\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/base.py\u001b[0m in \u001b[0;36m__repr__\u001b[0;34m(self, N_CHAR_MAX)\u001b[0m\n\u001b[1;32m    258\u001b[0m                 \u001b[0mvalid_params\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    259\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 260\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msub_params\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnested_params\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    261\u001b[0m             \u001b[0mvalid_params\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0msub_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/pprint.py\u001b[0m in \u001b[0;36mpformat\u001b[0;34m(self, object)\u001b[0m\n\u001b[1;32m    142\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mpformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m         \u001b[0msio\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_StringIO\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 144\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msio\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    145\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0msio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    146\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/pprint.py\u001b[0m in \u001b[0;36m_format\u001b[0;34m(self, object, stream, indent, allowance, context, level)\u001b[0m\n\u001b[1;32m    159\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_readable\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    160\u001b[0m             \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 161\u001b[0;31m         \u001b[0mrep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlevel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    162\u001b[0m         \u001b[0mmax_width\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_width\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mindent\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mallowance\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    163\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrep\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0mmax_width\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/pprint.py\u001b[0m in \u001b[0;36m_repr\u001b[0;34m(self, object, context, level)\u001b[0m\n\u001b[1;32m    391\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlevel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    392\u001b[0m         repr, readable, recursive = self.format(object, context.copy(),\n\u001b[0;32m--> 393\u001b[0;31m                                                 self._depth, level)\n\u001b[0m\u001b[1;32m    394\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mreadable\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    395\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_readable\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/utils/_pprint.py\u001b[0m in \u001b[0;36mformat\u001b[0;34m(self, object, context, maxlevels, level)\u001b[0m\n\u001b[1;32m    179\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    180\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_changed_only\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 181\u001b[0;31m             \u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_changed_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    182\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    183\u001b[0m             \u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdeep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/utils/_pprint.py\u001b[0m in \u001b[0;36m_safe_repr\u001b[0;34m(object, context, maxlevels, level, changed_only)\u001b[0m\n\u001b[1;32m    423\u001b[0m         \u001b[0mitems\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpprint\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_safe_tuple\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    424\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mitems\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 425\u001b[0;31m             krepr, kreadable, krecur = saferepr(\n\u001b[0m\u001b[1;32m    426\u001b[0m                 k, context, maxlevels, level, changed_only=changed_only)\n\u001b[1;32m    427\u001b[0m             vrepr, vreadable, vrecur = saferepr(\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/utils/_pprint.py\u001b[0m in \u001b[0;36m_changed_params\u001b[0;34m(estimator)\u001b[0m\n\u001b[1;32m     89\u001b[0m     estimator with non-default values.\"\"\"\n\u001b[1;32m     90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 91\u001b[0;31m     \u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdeep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     92\u001b[0m     \u001b[0mfiltered_params\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     93\u001b[0m     init_func = getattr(estimator.__init__, 'deprecated_original',\n",
      "\u001b[0;32m/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/base.py\u001b[0m in \u001b[0;36mget_params\u001b[0;34m(self, deep)\u001b[0m\n\u001b[1;32m    193\u001b[0m         \u001b[0mParameters\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    194\u001b[0m         \u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 195\u001b[0;31m         \u001b[0mdeep\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdefault\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    196\u001b[0m             \u001b[0mIf\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwill\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mparameters\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mthis\u001b[0m \u001b[0mestimator\u001b[0m \u001b[0;32mand\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    197\u001b[0m             \u001b[0mcontained\u001b[0m \u001b[0msubobjects\u001b[0m \u001b[0mthat\u001b[0m \u001b[0mare\u001b[0m \u001b[0mestimators\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'CRF' object has no attribute 'keep_tempfiles'"
     ]
    }
   ],
   "source": [
    "# 搭建一个CRF模型\n",
    "# 注意：报错'CRF' object has no attribute 'keep_tempfiles'不影响结果\n",
    "crf = CRF(\n",
    "    algorithm='lbfgs',   #训练算法\n",
    "    c1=0.1,    #L1正则化系数\n",
    "    c2=0.1,    #L2正则化系数\n",
    "    max_iterations=100,     #优化算法的最大迭代次数\n",
    "    all_possible_transitions=False\n",
    ")\n",
    "#使用crf模型对训练集进行训练\n",
    "crf.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(sentence):\n",
    "    '''\n",
    "        输出CRF预测的一个句子的BIO标注\n",
    "        入参：\n",
    "            sentence：待处理的句子\n",
    "        出参：\n",
    "            sent_bio：一个字典，字典的键为句子中的汉字，值为其对应的BIO标注\n",
    "    '''\n",
    "    # 获取输入句子的特征\n",
    "    fea = sentence2feature(sentence)\n",
    "    # 获取crf的预测值，格式为每个字的BIO标注列表\n",
    "    pred = crf.predict_single(fea)\n",
    "    # 将句子中的汉字与其对应的BIO标注进行配对并以字典结构存储\n",
    "    sent_bio = dict(zip(sentence,pred))\n",
    "    return sent_bio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'这': 'O',\n",
       " '是': 'O',\n",
       " '由': 'O',\n",
       " '于': 'O',\n",
       " '耳': 'B-BODY',\n",
       " '膜': 'I-BODY',\n",
       " '损': 'O',\n",
       " '伤': 'O',\n",
       " '导': 'O',\n",
       " '致': 'O',\n",
       " '的': 'O'}"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#使用predict函数对一个句子进行预测\n",
    "predict('这是由于耳膜损伤导致的')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出CRF模型能够有效的识别出这句话中的实体，接下来我们用CRF模型对我们的测试集进行预测。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "#获取测试集的预测值\n",
    "y_pred = crf.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用sklearn_crfsuite中自带的metrics包可对模型进行有效的评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B-BODY',\n",
       " 'I-BODY',\n",
       " 'B-SIGNS',\n",
       " 'I-SIGNS',\n",
       " 'B-TREATMENT',\n",
       " 'I-TREATMENT',\n",
       " 'B-EXAMINATIONS',\n",
       " 'I-EXAMINATIONS',\n",
       " 'B-DISEASES',\n",
       " 'I-DISEASES']"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#获取crf模型的全部标签\n",
    "labels = list(crf.classes_)\n",
    "#由于标签O过多，而我们对其他标签更感兴趣。为了解决这个问题，我们标签O移除。\n",
    "labels.remove('O')\n",
    "#查看除'O'外的全部标签\n",
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9417714544345397"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#计算除O之外的所有标签计算的平均F1分数。\n",
    "metrics.flat_f1_score(y_test, y_pred,\n",
    "                      average='micro', labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/utils/validation.py:72: FutureWarning: Pass labels=['B-BODY', 'I-BODY', 'B-SIGNS', 'I-SIGNS', 'B-TREATMENT', 'I-TREATMENT', 'B-EXAMINATIONS', 'I-EXAMINATIONS', 'B-DISEASES', 'I-DISEASES'] as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error\n",
      "  return f(**kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                precision    recall  f1-score   support\n",
      "\n",
      "        B-BODY      0.924     0.929     0.927      3145\n",
      "        I-BODY      0.911     0.938     0.925      5682\n",
      "       B-SIGNS      0.976     0.978     0.977      2273\n",
      "       I-SIGNS      0.976     0.976     0.976      2499\n",
      "   B-TREATMENT      0.916     0.826     0.869       316\n",
      "   I-TREATMENT      0.921     0.825     0.870      1501\n",
      "B-EXAMINATIONS      0.970     0.974     0.972      2824\n",
      "I-EXAMINATIONS      0.966     0.970     0.968      6254\n",
      "    B-DISEASES      0.863     0.726     0.789       208\n",
      "    I-DISEASES      0.828     0.731     0.776       746\n",
      "\n",
      "     micro avg      0.943     0.940     0.942     25448\n",
      "     macro avg      0.925     0.887     0.905     25448\n",
      "  weighted avg      0.943     0.940     0.941     25448\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#查看每个类别的预测情况\n",
    "print(metrics.flat_classification_report(\n",
    "    y_test, y_pred, labels=labels, digits=3\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BiLSTM-CRF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CRF模型还可与BiLSTM模型结合来解决实体识别问题，这样的好处是BiLSTM可以自动获取文本的特征，我们便不需要自己去定义特征，不需要再进行文本特征工程部分。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于BiLSTM-CRF的代码过于冗长，且实现这一代码并不是我们的重点，而仅做展示之用。所以我们把BiLSTM-CRF模型的实现细节均在BiLSTM_CRF.py中实现。这里仅展示部分关键部分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "from BiLSTM_CRF import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将word映射到id\n",
    "word2id = word_to_id(sentence_list)\n",
    "# 将label映射到id\n",
    "tag2id = tag_to_id(label_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'O': 0,\n",
       " 'B-TREATMENT': 1,\n",
       " 'I-TREATMENT': 2,\n",
       " 'B-SIGNS': 3,\n",
       " 'I-SIGNS': 4,\n",
       " 'B-EXAMINATIONS': 5,\n",
       " 'I-EXAMINATIONS': 6,\n",
       " 'B-BODY': 7,\n",
       " 'I-BODY': 8,\n",
       " 'B-DISEASES': 9,\n",
       " 'I-DISEASES': 10,\n",
       " '<unk>': 11,\n",
       " '<pad>': 12,\n",
       " '<start>': 13,\n",
       " '<end>': 14}"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#查看label与id的映射关系\n",
    "tag2id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LSTM模型训练的时候需要在word2id和tag2id加入PAD和UNK，如果是加了CRF的lstm还要加入<start>和<end> (解码的时候需要用到)。word2id的格式与tag2id的格式类似。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 按照与CRF模型划分训练集测试集时相同的比例和相同的随机数种子对sentence_list与label_list划分训练集合测试集\n",
    "x_train_lstmcrf,x_test_lstmcrf,y_train_lstmcrf,y_test_lstmcrf = train_test_split(sentence_list, label_list, test_size=0.3, random_state=2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 为每句话后加入一个\"<end>\" token\n",
    "x_train_lstmcrf,y_train_lstmcrf = prepocess_data_for_lstmcrf(x_train_lstmcrf,y_train_lstmcrf)\n",
    "x_test_lstmcrf,y_test_lstmcrf = prepocess_data_for_lstmcrf(x_test_lstmcrf,y_test_lstmcrf,test=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1, step/total_step: 10/14 71.43% Loss:701.0319\n",
      "Epoch 1, Val Loss:317.2419\n",
      "Epoch 2, step/total_step: 10/14 71.43% Loss:389.3850\n",
      "Epoch 2, Val Loss:238.6616\n",
      "Epoch 3, step/total_step: 10/14 71.43% Loss:318.3480\n",
      "Epoch 3, Val Loss:209.9835\n",
      "Epoch 4, step/total_step: 10/14 71.43% Loss:275.3966\n",
      "Epoch 4, Val Loss:182.5283\n",
      "Epoch 5, step/total_step: 10/14 71.43% Loss:240.5875\n",
      "Epoch 5, Val Loss:149.7029\n",
      "Epoch 6, step/total_step: 10/14 71.43% Loss:194.7677\n",
      "Epoch 6, Val Loss:121.5650\n",
      "Epoch 7, step/total_step: 10/14 71.43% Loss:159.2875\n",
      "Epoch 7, Val Loss:103.0207\n",
      "Epoch 8, step/total_step: 10/14 71.43% Loss:134.6785\n",
      "Epoch 8, Val Loss:86.8583\n",
      "Epoch 9, step/total_step: 10/14 71.43% Loss:113.2668\n",
      "Epoch 9, Val Loss:75.7414\n",
      "Epoch 10, step/total_step: 10/14 71.43% Loss:97.2053\n",
      "Epoch 10, Val Loss:67.8924\n",
      "Epoch 11, step/total_step: 10/14 71.43% Loss:87.2976\n",
      "Epoch 11, Val Loss:58.2037\n",
      "Epoch 12, step/total_step: 10/14 71.43% Loss:75.4258\n",
      "Epoch 12, Val Loss:52.3318\n",
      "Epoch 13, step/total_step: 10/14 71.43% Loss:67.4084\n",
      "Epoch 13, Val Loss:47.1158\n",
      "Epoch 14, step/total_step: 10/14 71.43% Loss:61.4081\n",
      "Epoch 14, Val Loss:43.9784\n",
      "Epoch 15, step/total_step: 10/14 71.43% Loss:56.6425\n",
      "Epoch 15, Val Loss:40.4003\n",
      "Epoch 16, step/total_step: 10/14 71.43% Loss:51.8829\n",
      "Epoch 16, Val Loss:36.8195\n",
      "Epoch 17, step/total_step: 10/14 71.43% Loss:47.5342\n",
      "Epoch 17, Val Loss:33.7759\n",
      "Epoch 18, step/total_step: 10/14 71.43% Loss:44.0274\n",
      "Epoch 18, Val Loss:31.4595\n",
      "Epoch 19, step/total_step: 10/14 71.43% Loss:41.1943\n",
      "Epoch 19, Val Loss:30.0022\n",
      "Epoch 20, step/total_step: 10/14 71.43% Loss:38.7682\n",
      "Epoch 20, Val Loss:28.4569\n",
      "Epoch 21, step/total_step: 10/14 71.43% Loss:37.0907\n",
      "Epoch 21, Val Loss:27.2490\n",
      "Epoch 22, step/total_step: 10/14 71.43% Loss:34.8743\n",
      "Epoch 22, Val Loss:25.6760\n",
      "Epoch 23, step/total_step: 10/14 71.43% Loss:33.6422\n",
      "Epoch 23, Val Loss:24.4413\n",
      "Epoch 24, step/total_step: 10/14 71.43% Loss:31.7653\n",
      "Epoch 24, Val Loss:23.5581\n",
      "Epoch 25, step/total_step: 10/14 71.43% Loss:30.8277\n",
      "Epoch 25, Val Loss:21.7880\n",
      "Epoch 26, step/total_step: 10/14 71.43% Loss:28.6456\n",
      "Epoch 26, Val Loss:20.8018\n",
      "Epoch 27, step/total_step: 10/14 71.43% Loss:27.6406\n",
      "Epoch 27, Val Loss:20.0341\n",
      "Epoch 28, step/total_step: 10/14 71.43% Loss:26.0955\n",
      "Epoch 28, Val Loss:19.9792\n",
      "Epoch 29, step/total_step: 10/14 71.43% Loss:25.7362\n",
      "Epoch 29, Val Loss:19.7573\n",
      "Epoch 30, step/total_step: 10/14 71.43% Loss:24.6754\n",
      "Epoch 30, Val Loss:18.9083\n"
     ]
    }
   ],
   "source": [
    "# 搭建一个BiLSTM_CRF模型\n",
    "# 注意：mac book air运行时间接近5小时\n",
    "model = BiLSTM_CRF_Model(vocab_size=len(word2id),out_size=len(tag2id),batch_size=64, epochs=30)\n",
    "# 在训练集上进行训练\n",
    "model.train(x_train_lstmcrf,y_train_lstmcrf,word2id,tag2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "#获取测试集的预测值\n",
    "y_pred_lstmcrf, _ = model.test(x_test_lstmcrf,y_test_lstmcrf,word2id,tag2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9263456090651558"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#计算BiLSTM-CRF模型除O之外的所有标签计算的平均F1分数。\n",
    "metrics.flat_f1_score(y_test_lstmcrf, y_pred_lstmcrf,\n",
    "                      average='micro', labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Applications/anaconda3/envs/mypy37/lib/python3.7/site-packages/sklearn/utils/validation.py:72: FutureWarning: Pass labels=['B-BODY', 'I-BODY', 'B-SIGNS', 'I-SIGNS', 'B-TREATMENT', 'I-TREATMENT', 'B-EXAMINATIONS', 'I-EXAMINATIONS', 'B-DISEASES', 'I-DISEASES'] as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error\n",
      "  return f(**kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                precision    recall  f1-score   support\n",
      "\n",
      "        B-BODY      0.914     0.914     0.914      3145\n",
      "        I-BODY      0.920     0.909     0.914      5682\n",
      "       B-SIGNS      0.966     0.946     0.956      2273\n",
      "       I-SIGNS      0.963     0.953     0.958      2499\n",
      "   B-TREATMENT      0.882     0.639     0.741       316\n",
      "   I-TREATMENT      0.901     0.807     0.851      1501\n",
      "B-EXAMINATIONS      0.965     0.945     0.955      2824\n",
      "I-EXAMINATIONS      0.955     0.949     0.952      6254\n",
      "    B-DISEASES      0.978     0.644     0.777       208\n",
      "    I-DISEASES      0.944     0.661     0.778       746\n",
      "\n",
      "     micro avg      0.941     0.912     0.926     25448\n",
      "     macro avg      0.939     0.837     0.880     25448\n",
      "  weighted avg      0.941     0.912     0.925     25448\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#查看BiLSTM-CRF模型每个类别的预测情况\n",
    "print(metrics.flat_classification_report(\n",
    "    y_test_lstmcrf, y_pred_lstmcrf, labels=labels, digits=3\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
